DOCUMENT RESUME 



ED 343 367 



EC 301 053 



AUTHOR 
TITLE 



INSTITUTION 

SPONS AGENCY 

PUB DATE 
CONTRACT 
NOTE 

PUB TYPE 

EDRS PRICE 
DESCRIPTORS 



IDENTIFIERS 



Baar, Richard; Ana Others 

Referral Consultant! An Expert system for Guiding 
Teachers in Referring Students for Special Education 
Placement. Final Report. 

Utah State Univ., Logan. Center for Persons with 
Disabilities. 

Special Educat on Programs (ED/OSERS) , Washington, 
DC. 

Aug 91 

H0023CH80069 
84p. 

Reports - Descriptive (141) 
NF01/PC04 Plus Postage. 

Bias; "Computer Uses in Education; Decision Making; 
•Disabilities; Elementary Education; "Expert Systems; 
•Referral; Special Education; student 
Characteristic r,; "Student Evaluation; Student 
Placement; Teacher Attitudes; Teacher Role 
Pre referral intervention 



ABSTRACT 

In light of evidence indicating that referral itself 
often predicts student placement, an expert system was designed to 
assist educators to reduce bias in the process of referring students 
with suspected disabilities. A preliminary review of the literature 
looks at teacher perceptions as a predictor of handicapping 
conditions, referral bias, and placement as a measure of handicapping 
conditions. The Referral Consultant expert system asks a series of 
questions which quickly assess, first, whether the student is an 
obvious candidate for special education referral, and then, if not, 
the student's academic achievement and learning problems, possible 
handicapping conditions to account for the learning problems, other 
difficulties possibly accounting for the learning problems, and 
whether pre- referral interventions have been attempted. The output is 
a report describing its conclusions as to whether the student should 
be referred for special education assessment. The system was 
evaluated in 18 elementary schools. Implementation of the system had 
no statistically significant impact on the percentage of students 
referred for special education, or the ratio of students placed to 
students assessed. Additionally, variables predicting referral for 
special education assessment were not consistent. Another report 
detailing the Referral consultant logic, questions, and output is 
attached. (21 references) (DB) 
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REFERRAL CONSULTANT 



Final Report 
Problem Statement 



Infcgggasfcion 

Techniques used by schools to conduct referral and placement 
procedures have a significant impact in determining who will 
receive special education services. Ideally, referral procedures 
should include specific criteria to be considered in deciding if 
a student should be assessed. Those criteria should be accurate 
predictors of the presence of a handicapping condition. Such a 
set of predictive referral criteria would ensure that costly 
assessment resources are used judiciously. Similarly, assessment 
procedures should be tied to specific state and federal 
guidelines that lead to the identification of students with 
specific handicapping conditions and placement in special 
education programs that foster the development of those students. 

In recent years, much research has been directed toward 
investigating the special education referral/assessment/placement 
process. However, the available data provide less than a 
complete understanding of that process. The following review is 
aimed at summarizing current literature describing the 
referral/assessment/placement process. This review will lead to 
the identification of important questions vhich have not been 
considered. 



Teacher Perceptions as a Predictor o f Handicap ping Conditions 

As noted, referral data should predict the presence of 
handicapping conditions. Unfortunately, referral procedures vary 
widely from state to state and school district to school 
district. There is a paucity of data on what referral 
information is typically collected and how well that information 
predicts the presence of handicapping conditions. For the most 
part, referral for special education assessment seems to be based 
on teacher concerns and perceptions that are not well specified. 
For example, Ysseldyke, Christenson, Pianta, and Algozzine (1983) 
in studying teachers' reasons for referral concluded they were 
generally stated in vague and nebulous terms. 

Recently, Gerber and Semmel (1985) have suggested that 
teacher perceptions are good predictors of placement in special 
services. These researchers had ten teachers in a rural school 
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serving kindergarten through eighth grade students, rate 266 
students on a variety of characteristics. During the school 
year, 11 students were referred for special education and 9 
ultimately received special education or other remedial services. 
Further, it was found that teacher ratings of students, 
particularly those on need for supervision, academic potential, 
and ease of teaching, were predictive of which students would be 
referred* Referred students generally had the lowest ratings in 
their classes. 

Data supplied by Shinn, Tindal, and Spira (1987) also 
suggest that teacher perceptions are predictive of the need for 
special services. These researchers established reading norms 
for a large midwestern city. They then used curriculum-based 
assessment procedures to assess reading achievement in 570 second 
through sixth grade students referred for difficulty in reading. 
Referred students were consistently found to have lower than 
average reading achievement. 



Although both Gerber and Semmel's (1985) and Shinn, Tindal, 
and Spira 's (1987) studies suggest that teachers' perceptions nay 
be good predictors, both qualify their results by presenting data 
indicating that such perceptions are less than perfect predictors 
and are biased. In the former study, although referred students 
tended to have the lowest teacher ratings in their classes, their 
ratings were not necessarily the lowest ratings in their schools. 
Further, having a low rating did not always predict referral and 
when teachers referred students who were not rated lowest, 
mitigating or biasing factors were identified. These included: 
1) the need for a high level of supervision, 2) having several 
previously identified handicapped students in the classroom, and 
3) having many lew rated students in the classroom and high 
variability in class ratings. Similarly, Shinn, Tindal, and 
Spira (1987) found sex and ethnicity to be biasing factors in 
referral, with larger percentages of male and black students 
being referred than would hare been predicted from population 
base rates of reading achievement. 

Numerous other studies have also examined bias in the 
referral process. These can be divided into those that examined 
student, teacher, class, and system characteristics. 

Student characteristics . Student characteristics that have 
been shown to bias the referral process include: 1) sex * males 
are more likely to be referred than females (Mendelsohn & 
Joinings, 1986; Sevick 6 Ysseldyke, 1986? Kelly, Bullock & Dykes, 
1977), 2) ethnicity - black and low socio-economic status (SES) 
Mexican Americans are more likely to be referred than whites 
(Argulewicz & Sanchez, 1983; Mendelsohn & Jennings, 1986), 3) 
attractiveness - unattractive students are more likely to be 
referred than attractive students (Ross & Salvia, 1975; 
Ysseldyke, Algozzlne, Regan, & McGue, 1981) and SES - low SES 



students are store likely to be referred then high SES students 
(Argulewicz & Sanchez, 1983? Ysseldyke, Algozzine, Regan, & 
McGue, 1981). 

Teach er characteristics . Teacher characteristics that have 
been shown to bias the referral process include: 1) sex - female 
teachers are store likely to refer students than stale teachers 
(Kelly, Bullock & Dykes, 1979; Sandler, 1980), 2) ethnicity - 
teachers are more likely to ref>?r students of ethnicity different 
from the teacher's (Kelly, Bullock & Dykes, 1977; Tobias, Cole, 
Zibren, & Bodlakova, 1982) , 3) teacher perceptions of the 
professionals receiving their referrals - teachers with positive 
perceptions are sore likely to refer students for assessment 
(Christenson, Ysseldyke, & Algozzine, 1982) , 4) teacher opinions 
regarding mainstreaming - teachers with low opinions of 
mainstreaming are more likely to refer students for assessment 
(Smart, Wilton, & Keeling, 1980) and 5) marital status - single 
teachers are more likely to refer students than married teachers 
(Smart, Wilton, & Keeling, 1980). 

Classroom factors . Classroom factors that have beer, shown 
to bias the referral process include: 1) number of mainstreawed 
students in the class - students from classes with relatively 
high numbers of mainstreamed students are more likely to be 
referred (Smart, Wilton & Keeling, 1980; Riffle, 1985) and 2) 
class size - students are more likely to be referred from large 
classes (Christenson, Ysseldyke, & Algozzine, 1982) . 

System Characteristics . Finally, regarding system 
characteristics, Christenson, Ysseldyke, and Algozzine (1982) 
have noted that advocacy groups, confusion over state and federal 
guidelines, and parent pressure impact the referral process. 
Further, several studies have shown that the availability of 
pre-referral consultation impacts the referral process - teachers 
provided such consultation are less likely to refer (Ritter, 
1978; Gut kin, singer, & Brown, 1980). 

Taken together the data on bias in referral seems to 
indicate that teacher perceptions of students that may be 
handicapped and in need of referral are influenced by many 
biasing factors. However, none of these factors should have a 
place in the decision making process. 



Placement as a Measure of Handicapping Conditions 

Data supplied by Ysseldyke and his colleagues and 
Hofmeister, Ferrara, and Likens (personal communication) raise 
questions about all studies that have looked at placement as a 
measure of the degree to which referred students exhibit 
handicapping conditions. Christenson, Algozzine, and Ysseldyke 
(1982) found the fact of referral to be the most important factor 
in determining special education placement. In a national survey 
they discovered: 1) from three to six percent of the school age 
population is referred for special education services each year, 
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2) 92 percent of those referred are tested, and 3) 73 percent of 
those tested are qualified for special education services. 
Further, Ysseldyke, Chris tenson, Pianta, and Algozzine (1983) 
found that teachers making referrals generally wanted testing and 
placement in special education. It appears that the fact of 
referral itself rather than any basis on which it is made is most 
predictive of special education placement. Further, Algozzine 
and Ysseldyke (1981) report that, at least in the case of 
learning disabled students, placement decisions are probably, to 
a large degree, unreliable. They found that when placement 
decision makers were presented with assessment data indicating 
normal performance, more t*han half the time, they reported they 
would have qualified the student for special education services. 
Also, ysseldyke, Algozzine, Richey, and Graden (1982) found 
little relationship between assessment data and placement team 
decisions. Support for the unreliability of placement decisions 
also comes from research conducted by Hofmeister, Ferrara, and 
Likens (personal communication) . As part of a current study, 
these researchers systematically examined files of students 
classified as learning disabled in ten Utah school districts to 
determine if the assessment data they contained could justify an 
LD classification based on federal and State of Utah regulations. 
They found that the classification of approximately half of the 
students could not be justified, results very similar to those 
obtained by Algozzine and Ysseldyke (1981) . Thus, referral 
itself, seems to predict placement and the validity of many 
special education placements is questionable. 
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Response to Problem 



In response to the need to provide educators with tools to 
improve the referral process, the Technology Division (TD) of the 
Center for Persons with Disabilities (CPD) at Utah State 
University (USU) proposed to develop an expert system computer 
program, Referral Consultant. The system would contain objective 
criteria for deciding whether or not to refer students for 
special education assessment and provide a second opinion 
regarding the appropriateness of referrals. It was anticipated 
that implementation of the system would reduce bias in the 
referral process and improve the accuracy and efficiency of the 
referral/assessment/placement process . 



Expert Systems 

Recent advances in artificial intelligence (AI) computer 
technology, specifically expert systems technology, offer the 
potential for improving the referral and placement procedures of 
public schools. 

Artificial intelligence. Professionals in the field of AI 
develop computing systems that attempt to replicate human 
characteristics, such as understanding, learning, reasoning, and 
problem solving (Barr & Feigenbaum, 1981) . 

Expert systems . Currently, the most commonly available AI 
product is the expert system. Typically, expert systems attempt 
to replicate the decision-making or problem-solving processes 
used by persons who are knowledgeable and experienced in a 
particular field (Hofmeister & Ferrara, 1987). Such expert 
systems engage users in a dialogue which in many ways, parallels 
expertise in a specific area. The system questions the user to: 
1) pinpoint the problem, and 2) provide the information needed by 
the expert system to suggest a solution (Stefik, Aikins, Balzer, 
Benoit, Bimbaum, Hayes-Roth, & Sacerdoti, 1983) . Solutions are 
generated through the interaction of two expert system 
components, an inference engine and a knowledge base. The 
knowledge base contains the knowledge needed to make a decision 
in a particular problem domain. The inference engine is a 
computer program designed to manipulate this knowledge base in a 
fashion which produces accurate and appropriate problem 
resolutions . 
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Referral Consultant Expert System 



System Development 

The Referral Consultant (RC) expert system was developed 
during the first year of the project by two task forces. Task 
force one was composed of special educators from Utah's Granite 
school district; task force two was composed of special educators 
from the Blackhills Special Services Cooperative in South Dakota. 

Project staff made a presentation to each task force 
outlining problems research had identified in the 
referral/assessment/placement process. The presentation pointed 
out that referral for special education assessment was influenced 
by a variety of biasing factors, that the fact of referral was 
the best predictor of placement in special education, and that 
many special education placements were probably inappropriate. 
Following the presentation, each task force was asked to address 
the question, on what basis should the decision to refer students 
for special education be made? This generated lively debate 
among both task forces leading to a list of factors that should 
be considered in making referrals and rules for how those factors 
should fre considered. 

Based on the decisions of the task forces at their initial 
meeting project staff developed a logic document describing how 
the RC expert system would work and programmed an initial version 
of the system. The logic-document was sent to all members of the 
task forces in advance of a second meeting where the initial 
version of RC was demonstrated. During the demonstration, RC's 
logic was discussed in detail and task force members pointed out 
weaknesses and made suggestions for changes. 

Based on input obtained at the second meeting of the task 
forces, project staff rewrote the logic document, reprogrammed 
the RC expert system, and demonstrated the revised system. This 
process of review and revision was continued through several 
cycles until both task forces were satisfied with RC. 

Description of Referral Consultant 

A complete description of the final logic decided on for 
Referral Consultant is contained in Appendix A. Following, is a 
brief description to assist the reader. 

1. Referral Consultant first asks a series of questions to 
determine if a student should obviously be referred for special 
education assessment. For example, does the student have little 
or no ability to speak or understand language? If it concludes 
the student should obviously be referred, it sends a message to 
do so. 

2. For students who are not obvious referrals, RC first 
compares what the target student can do academically to what the 
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average student in his/her class can do. This information is 
used to arrive at a confidence that the student has learning 
problems or adverse effects on educational performance. The 
confidence is expressed as a number from 1 to 100 and is similar 
to a percent. For example, RC might arrive at a confidence 
factor (cf) of 80, indicating it is 80 percent confident that the 
student is experiencing learning problems. Unless information 
provided on the home environment, culture, and/or educational 
history indicates otherwise, this is also RCs confidence that 
the student should be referred for special education assessment. 

3. After determining a student is experiencing learning 
problems^ RC goes on to ask questions that lead it to conclusions 
regarding whether those learning problems can be attributed to: 
1) behavioral/ social -emotional , 2) sensory (vision or hearing), 
or 3) physical health difficulties. Separate confidence factors 
indicating how confident RC is that learning problems are due to 
behavioral or physical difficulties are assigned. For example, 
RC might conclude that a student's learning problem could be 
accounted for by behavioral/social-emotional difficulties at a cf 
of 95. This would indicate RC is 95 percent confident learning 
problems were due to behavioral/social-emotional difficulties. 
Observations suggesting the student may have vision or hearing 
problems are also noted. 

4. Next, RC asks questions that lead it to conclusions 
regarding whether a student's learning problems can be accounted 
for by non-special education difficulties. The three areas 
considered are the home environment, cultural differences, and 
educational history. Separate confidence factors indicate how 
confident RC is that learning problems are due to difficulties in 
these areas. 

5. Finally, RC asks if two "quality" pre-referral interventions 
have been tried to help the student. If this has not been done, 
RC sends a message that its other conclusions are tentative and 
that appropriate pre-referral interventions should be tried. 

RCs final output is a report describing its conclusions 
with regard to whether the student should be referred for special 
education assessment. The report can be displayed on the 
computer monitor or printed for a permanent record. In addition, 
it may serve as the formal referral form on the student if 
warranted. An example of a RC report is contained in Appendix B. 
The report is divided into nine sections: 

1. Section one presents demographic information on the 
student; consultation date, name, school, grade, age, teachers 
name, and a space to fill in the date the student's parents were 
notified of the referral. 

2. Section two presents RCs conclusions regarding whether 
the student has learning problems. An overall confidence level 
is presented on a scale from 0 (completely confident the student 
does not have learning problems) to 100 (completely confident the 
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student does have learning problems) . Unless information on the 
student's home environment, culture, or educational history 
indicates that learning problems result from these factors, the 
confidence factor presented in this section represents the 
confidence the user can have that the student should be referred 
for special education assessment. In addition, the percent of 
skills the student being considered for referral has mastered 
compared to same sex peers in his/her class is presented for four 
academic areas; mathematics, reading, science, and social 
studies . 

3. Section three presents RCs conclusion regarding how 
confident the user can be that learning problems result from 
behavioral/social-emotional difficulties. Also, presented are 
specific behavior problems (e.g. stealing, nightmares, etc.) that 
led RC to its conclusion. 

4. Section four presents RCs conclusion regarding how 
confident the user can be that learning problems result from 
physical/health problems. The section also contains a listing of 
diagnosed health problems, physical symptoms observed in the 
student, and medications taken for behavioral control or 
physical/health problems. Finally, observations that may indicate 
that the student has vision or hearing problems are listed. 

5. Section five presents RCs conclusion regarding how 
confident the user can be that learning problems result from 
problems in the home environment. Specific family problems that 
may be impacting the student are listed (e.g. divorce, loss of 
family income, etc.). 

6. Section six presents RCs conclusion regarding how 
confident the user can be that learning problems result from 
cultural differences. Specific information provided includes an 
estimate of the student' English language proficiency compared to 
same age peers, a statement of the primary language spoken in the 
student's non-school environment, the number of years of English 
as a second language instruction provided the student, the 
student's ethnic background, the percentage of students in the 
student's school belonging to the same ethnic background, and the 
number of years the student has resided in the United States. 

7. Section seven presents RCs conclusion regarding how 
confident the user can be that learning problems result from an 
"unusual" educational history, specific problems the student has 
experienced are presented (e.g. frequent change of schools, poor 
school attendance, etc. ) . 

8. Section eight lists pre-referral interventions that were 
attempted to ameliorate the student's school difficulties. 

9. Section nine provides a complete listing of skills the 
student being considered for referral has and has not mastered 
compared to the average same sex student in the class by four 
academic areas; math, reading, science, and social studies. 
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Research Methodology 



Participating Schools 

Eighteen elementary schools cooperated in assessing the 
impact of RC on the referral/ assessment/placement process. Ten 
schools were from Utah's Granite school district; the other eight 
were from the B." - fLUs Special Services Cooperative in South 
Dakota. 

Research Design 

The basic research design used in assessing the impact of RC 
is illustrated in Figure 1. Half the schools in Utah were 
randomly assigned to an experimental group; the other half were 
assigned to a control group. In South Dakota, five schools were 
randomly assigned to an experimental group; three were assigned 
to a control group. During project year one, baseline data were 
collected at all schools. In year two, the experimental schools 
were provided with the RC expert system and asked to use it as an 
aid in determining whether students should be referred for 
special education assessment. This design allowed for comparison 
of differences on a variety of outcome measures from baseline 
(year 1) to treatment (year 2) and between the experimental and 
control groups. 



Year 1 
(Baseline) 

State 
Utah 



Year 2 
(Referral Consultant) 



State 
South Dakota 



Experimental 
Group 
Control 



Experimental 
Group 
Control 



Figure 1. Research design. 



Procedure 

To facilitate data collection, a data coordinator was 
identified in each school in each state. During year one, data 
coordinators assisted in the collect! ion of baseline data aginst 
which the effects of implementing RC in year two could be 
measured. 

At the beginning of year two a training workshop was held in 
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each state for the data coordinators from schools implementing 
RC. Coordinators were provided with necessary computer equipment 
and an Easy Reference Guide (see Appendix C) to using RC. They 
were introduced to the program and provided practice in running 
consultations. Coordinators were also trained in the use of DOS. 

Data cooridnators at schools implementing RC were instructed 
to run the first RC consultation with each referring teacher. 
Thereafter, referring teachers could elect to run successive 
consultations for other students independently. As it turned 
out, in South Dakota an RC consultation was run on all students 
who were referred for special education assessment. In Utah 
running an RC consultation on a referred student was made 
optional by the district special education director. Referring 
teachers and data coordinators could choose to use RC or not. 
Many teachers and data coordinators in Utah ran RC consultations 
based on the number of referrals that had to be processed. 
Running an RC consultation was reportedly time-consuming. 
Therefore, when pressure to process referrals was high (fall and 
spring), many Utah teachers elected to bypass RC for purposes of 
expediency. Data collected during year two indicated that RC 
consultations were rur on 42% of student referred for special 
education assessment. 

Results 

Impact on Referral 

Data were collected on the number of students referred for 
special education assessment each school. The raw data were 
converted to the percent of students referred per 100 students in 
the school via the formula , "number of students in the school 
referred for special education assessment/number of students in 
the school". Table 1 presents the mean percent of students 
referred in the experimental and control groups in Utah and South 
Dakota during years one and two. Associated standard deviations 
are also presented. 

Table 1 

Mean Percent of Students Referred for Special Education 

Assessment 



IsilJ Twr I 

Utah South Dakota Utah South Dakota 

Experimental Control Exoarfwantel Control Experimental Control gSBjfciflmillJ gop^ol 

M 3.78 4.06 2.81 2.31 2.71 3.34 1.72 1.50 

SO 1.21 2.SS 2.99 1.S2 1.46 1.15 1.00 1.36 
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The data were analyzed by means of two way analyses of variance 
(ANOVA) . The specific ANOVA was a 2 (states - Utah vs. South Dakota) 
x 2 (groups - experimental vs. control) . Tables 2 and 3 present the 
results of the ANOVAs for years one and two respectively. The 
analyses yielded only one significant effect. The main effect for 
states was significant at greater than the .05 level in year two. 
Utah referred a significantly greater percentage of students in year 
two than did South Dakota. Implementation of Referral Consultant in 
year two had no statistically significant impact on the percent of 
students referred for special education assessment. 

Table 2 

ANOVA - Referrals/100 Students - Year 1 



Sum of 

Source of Variation Squares 

Main Effects 7.543 

STATE 7.486 

GROUP .011 

2 -way Interactions .660 

STATE X GROUP .660 

Explained 8.203 

Residual 72.132 

Total 80.335 



Mean Significance 



EE 


Square 


£ 


Of F 


2 


3.771 


.732 


.498 


1 


7.486 


1.453 


.248 


1 


.011 


.002 


.963 


1 


.660 


.128 


.726 


1 


.660 


.128 


.726 


3 


2.734 


.531 


.669 


14 


5.152 






17 


4.726 
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Table 3 

AMOVA - Referrals/loo Students - Year 2 



Source of Variation 

Main Effects 
STATE 
GROUP 

2 way Interactions 
STATE X GROUP 

Explained 

Residual 

Total 



Sub of 




Mean 


Scruares 


BE 


Scruare 


8.900 


2 


4.450 


8.053 


1 


8.053 


.312 


1 


.312 


.782 


1 


.782 


.782 


1 


.782 


9.682 


3 


3.227 


21.452 


14 


1.532 


31.134 


17 


1.831 



Significance 



£ 




2.904 


.088 


5.256 


.038 


.203 


.659 


.511 


.487 


.511 


.487 


2.106 


.145 



There was an overall tendency for schools in all groups to 
refer lower percentages of students in year two. Interestingly, 
this tendency was greater in the experimental than in the control 
groups. Utah's control group referred .72 percent fewer students 
in year two than in year one, while South Dakota's control group 
referred .81 percent fewer students. In contrast, Utah's 
experimental group referred 1.07 percent fewer students in year 
two than in year one; South Dakota's referred 1.09 percent fewer. 
Thus, the decrease in the percentage of students referred from 
year one to two was .35 percent greater in Utah's experimental 
group than in its control group and .28 percent, greater in South 
Dakota's experimental group than in its control group. However, 
as noted, this effect was not statistically significant. 

Impact on Placement 

Data were also collected on the number of students placed in 
special education in each school. The raw data were converted to 
the percent of students placed per 100 students in the school via 
the formula, "number of students in the school placed in special 
education/number of students in the school n . Table 4 presents the 
mean percent of students placed in the experimental and control 
groups in Utah and South Dakota during years one and two. 
Associated standard deviations are also presented. 
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Table 4 

Mean Percent of Students Placed in Special Education 



IIILJ lSAL-2 

Ulftb south o.kof jujU Stt^ ttlHnM — 

Experimental Control £xp«H»entat Control EnWffftnttl C9fl*r9t EMWllWtttl SSDiZ&L 

M 2.19 2.16 f .01 1.14 1.77 1.77 .83 .61 

SD .81 1.56 1.05 .59 1.06 .63 .84 .37 



Aawith referrals, the data vere analyzed by means of two way 
ANOVAs.VThe specific ANOVA was again a 2 (states - Utah vs. South 
Dakota) V 2 ( groups - experimental vs. control). Tables 5 and 6 
present xhe^results of the ANOVAs for years one and two 
respectively. THer^aaalyses yielded only one significant effect. The 
main effect for states was significant at greater than the .05 level 
in year two. Utah placed a significantly greater percentage of 
students in year two than did South Dakota, implementation of 
Referral Consultant in year two had no impact on the percent of 
students placed in special education. 

Table 5 

ANOVA - Placements /100 Students - Year 1 





Sum of 




Mean 


Significance 


Source of Variation 


Sauares 


EE 




I 


ot F 


Main Effects 


5.550 


2 


2.775 


2.283 


.139 


STATE 


5.411 


1 


5.411 


4.452 


.053 


GROUP 


.006 


1 


.006 


.005 


.944 


2-way Interactions 


.031 


1 


.031 


.025 


.876 


STATE X GROUP 


.031 


1 


.031 


.025 


.876 


Explained 


5.580 


3 


1.860 


1.530 


.250 


Residual 


17.016 


14 


1.215 






Total 


22.597 


17 


1.329 
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Table 6 

anova - Placement/ 100 Students - Year 2 



source of Variation 


Sun of 


EE 


Mean 
Souare 


Z 


Significance 
OJf F 


Main Effects 
STATE 

GROUP 


4.672 
4.667 
. 040 


2 
1 
1 


2.336 
4.667 
.040 


3.568 
7.128 
♦ 061 


.056 
.018 
.808 


2 -way interactions 
STATE X GROUP 


.050 
.050 


1 
1 


.050 
.050 


.076 
.076 


.787 
.787 


Explained 


4.722 


3 


1.574 


2.404 


.111 


Residual 


9.166 


14 


.655 






Total 


13.888 


17 


.817 







As with referrals, there was an overall tendency for schools 
in all groups to place lower percentages of students in year two. 
However, there was no tendency for the decreases to be greater in 
the experimental than in the control groups. Utah's experimental 
group placed .42 percent fewer students in year two than in year 
one, while its control group placed .39 percent fewer. South 
Dakota's experimental group placed .18 percent fewer students in 
year 2 than in year one, while its experimental group placed .53 
percent fewer. 

Impact on Referral Efficacy - 
Placement/Referral Ratio 

The percent of students referred for special education 
assessment who were placed in special education in each school was 
taken as one measure of referral efficacy. The rationale here was 
that an efficient referral system would refer mostly students who 
were ultimately placed in special education and few who were 
assessed and not placed. Thus the system would discriminate with a 
high degree of accuracy those students in need of special education 
services. Table 7 presents the mean percent of students placed for 
the experimental and control groups in Utah and South Dakota during 
years one and two. Associated standard deviation are also presented. 
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Table 7 

Percent of Students Referred Who were Placed in Special Education 



t«t i IM£_2 

"fh south a«kot» mh mih Mtott — 

Experimental Control Experimental Control ExpgHiiwntal fionjroj. Experimental SsBUSi 

M 38 53 38 57 65 54 38 49 

SO 14 16 23 23 10 10 29 18 



Again, the data were analyzed by means of 2 (states) x 2 
(groups) ANOVAs. Tables 8 and 9 present the results of the 
ANOVAs for years one and two respectively. No significant 
effects were yielded. Implementation of RC in year two had no 
impact on the percent of students placed in special education. 

Table 8 

ANOVA - Pe rcent of Students Place - Year 1 



Source of Variation 

Main effects 
STATE 
GROUP 

2 -way Interactions 
STATE X GROUP 

Explained 

Residual 

Total 



Sum of 




Mean 






signif 


Squares 


DF 


Sauare 






9t 


638.218 


2 


319.109 


• 


882 


.436 


439.839 


1 


439.938 




216 


.289 


128.884 


1 


128.884 


* 


356 


.560 


628.654 


1 


628.654 


1. 


738 


.209 


628.654 


1 


628.654 


1. 


738 


.209 


1266.873 


3 


422.291 


1. 


167 


.357 


5064.561 


14 


361.754 








6331.434 


17 


372.437 
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Table 9 

ANOVA - Pe rcent of Students Placed - Year 2 



Sun of Mean . Significance 



Sours© Qf variation 


squares 


CE 


square 




z 


of F 


Main Effects 


1299.967 


2 


649. 


984 


1. 


853 


.193 


£*ffl^ III LI 

STATE 


±2t*0 • 031 


l 


1298. 


851 


•> 




• U / D 


GROUP 


11.947 


1 


11. 


947 


• 


034 


.856 


2 -way Interactions 


518.883 


1 


518. 


883 


1. 


480 


.244 


STATE X GROUP 


518.883 


1 


518. 


883 


1. 


480 


.244 


Explained 


1818.851 


3 


606. 


284 


1. 


729 


.207 


Residual 


4909.621 


14 


350. 


687 








Total 


6728.472 


17 


395. 


792 









Impact on Referral Efficacy - 
Assessment Time 

Assessment time was taken as a second index of referral 
efficacy. Members of multidiciplinary assessment teams at all 
participating schools were asked to track the amount of time they 
spent administering tests, observing students, writing reports, 
attending meetings, and in other assessment related activities. 
Table 10 presents the mean hours of assessment time per student 
for the experimental and control groups in Utah and South Dakota 
during years one and two. Associated standard deviations are also 
presented. 

Table 10 

Mean Hours of Assessment Time for Students Assessed for Special 
Educatio n 



South Dakota 



Utah 



South Dakota 



Experimental Control Experimental Control ExpeHffgrNftt iSSlISL ErW'ffWnfl £2QI££l 



8.81 
9.69 



13.40 
15.95 



10.23 
9.98 



28.26 
26.75 



5.79 
3.67 



6.23 
4.86 



6.81 
4.35 



7.64 
4.76 



The assessment time data were analyzed via a three way analysis 
of variance. Specifically, the analyses was a 2 (state) x 2 (group) 
x 2 (year) ANOVA. Table 11 presents the ANOVA results. As can be 
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noted, all main and interaction effects were significant 
Table 11 

ANQVA - A ssessment Tine 





Sum of 




Mean 




Significance 


source 5Ui vari^ion 


RmiAfOQ 


DP 


Souars 


F 




Of F 


Main Effects 


13246.218 


3 


4415.406 


32. 


574 


0.0 


O lAl£t 


9877 8Q£ 


1 


2877 896 


21. 




Ann 


GROUP 


3992.497 


1 


3992*497 


29. 


454 


• 000 




7211 &AA 


1 


/ A XX • WWW 


53 - 




nnn 




JU7 J « O Wa# 


3 


1698 .621 


12 . 


531 


.000 


STATE X GROUP 


2042.715 


i 


2042.715 


15. 


070 


.000 


STATE X YEAR 


1134.985 


i 


1134.985 


8. 


373 


.004 


GROUP X YEAR 


2209.857 


i 


2209.857 


16. 


303 


.000 


3 -way Interactions 


1287.226 


i 


1287.226 


9. 


496 


.002 


STATE X GROUP X 


YEAR 1287.226 


i 


1287.226 


9. 


496 


.002 


Explained 


19629.306 


7 


2804.187 


20. 


688 


0.0 


Residual 


90681.978 


669 


135.549 








Total 


110311.284 


676 


163.^82 









The main effect for state was significant at greater than the 
student on assessment (x - 13.22 hrs. /student) than did Utah (x - 
8.97 hrs. /student ) . The group main effect was also significant at 
greater than the .001 level of confidence, overall, more time was 
devoted to assessing students in the control group (x ■ 12.10 
hrs. /student) than was devoted to assessing students in the 
experimental group (x • 7.97 hrs. /student) . Finally, the year main 
effect was also significant at greater than the .001 level of 
confidence. Overall, more time was spent assessing students in year 
one (x = 12.81 hrs. /student) than in year two (x « 6.31 
hrs. /student) . 

The assessment time state x group interaction was significant 
at greater than the .001 level of confidence. This interaction is 
shown in Figure 2. In Utah, on average 7.55 hours were spent 
assessing students in the experimental group. On average, 10.14 
hours were spent assessing students in the control group, 2.59 hours 
more than for students in the experimental group. In South Dakota, 
more time was spent assessing students in the control than in the 
experimental group and the difference was relatively more than in 
Utah. On average, 8.87 hours were spent assessing students in the 
experimental group; 19.89 hours were spent assessing control group 
students, 11.02 hours more than for the experimental group. 
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Experimental control 



Figure 2. Assessment time interaction - state x group. 

The state x year assessment time interaction was also 
significant at the .004 level of confidence. This interaction is 
illustrated in Figure 3. In Utah, on average, 11.25 hours were 
spent assessing students in year one. This decreased by 5.21 hours 
to 6.04 hours per student in year two. South Dakota's assessment 
time per student also decreased from year one to year two and was 
relatively greater than the decrease for Utah. In year one and 
average of 17.27 hours per student was spent in assessment 
activities. This decreased by 10.13 hours to 7.14 hours per student 
in year two. 
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— Utah 

— SouthDttetA 



11.2 



Year 1 Year 2 

Figure 3. Assessment time interaction - state x year. 
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The assessment time group x year interaction was significant at 
greater than the .001 level of confidence. This interaction is 
shown in Figure 4. in year one an average of 9.27 hours per student 
was spent in assessment. This decreased by 3.17 hours per student 
to 6.10 hours in year two. The mean hours of assessment per student 
also decreased from year one to year two for the control group and 
by relatively more than for the experimental group. In year one, an 
average of 16.59 hours per student were spent in assessment. This 
decreased by 11.50 hours to 5.09 hours in year two. 

Lbqmm 

17 4 16.59 
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Year 1 Year 2 

Figure 4. Assessment time interaction - group x year. 

The three way interaction, state x group x year, for assessment 
time was also significant at the .002 level of confidence. This 
interaction is shown in Figure 5. As the figure shows, the mean 
hours of assessment time decreased for both groups in both states. 
Decreases were relatively greater for the control than experimental 
arouos. and relatively greater in South Dakota than in Utah. 
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Figure 5. 



Year 1 

Assessment time interaction 



Year 2 
- state x group x year. 
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impact on Bias 



For each referred student, data were collected on a variety of 
teacher, student, and class characteristics previous research had 
show to bias the referral process. The sane data vere also collected 
on randomly selected students matched with the referred students for 
sex and grade level who had not been referred for special education 
in Appendix D. The characteristics of interest were as follows: 

1. Teacher Gender - male or female 

2. Teacher Ethnicity - white or nonwhite 

3. Teacher Marital Status - married or unmarried 

4. Teacher Has Children - has children or has no children 

5. competence Professionals Receiving Referral - rated on a 
scale from 1 = completely incompetent to 5 « extremely 
competent . 

6. Appropriateness of Mainstr eamina Mild/Moderate Disabled 
Students - rated by the teacher on a scale from 1 = 
disagree to 5 ■ agree. 

7. A ppropriateness of Mainstreamina Severe Disabled Students 
- rated by the teacher on a scale from 1 = disagree to 5 
= agree. 

8. Class Potential - teacher rated each student in his/her 
class on a scale from 1 = low potential tc 5 = 
high potential. 

9. variabilit y Potential - the standard deviation of the 
ratings described in item eight. 

10. Mainstream ed Students - the percentage of students in the 
class who were mainstreamed students. 

11. Class Size - the number of students in the class. 

Stepwise discriminant analyses were run on the above 
variables to determine which variables best predicted referral 
for special education assessment. Separate analyses were run for 
the experimental and control groups in Utah and South Dakota in 
both year one and year two. Thus, eight discriminate analyses 
were run in all. 

Table 12 presents the standardized coefficients for each of the 
discriminant analyses. As the table shows, none of the variables 
were always included among those that best predicted referral . 
Teachers' perceptions of the appropriateness of mainstreaming 
severe/moderate students and their perceptions of the 
appropriateness of mainstreaming severe students were the variables 
most often found to be predictive of referral. Each was found to be 
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an important discriminator in five of the eight analyses, although 
not necessarily in the same analyses. Interestingly, the direction 
of prediction was not consistent across the analyses in which these 
variables were shown to he important predictors. In four of the 
analyses in which teacher perceptions of the appropriateness of 
mainstreaming mild/moderate students appeared as important 
predictors high rating of appropriateness were associated with 
referral. However, in one analysis low ratings were associated with 
referral. Similarly, in three of the five analyses in which teacher 
perceptions of the appropriateness of mainstreaming severe students 
appeared as important predictors high ratings of appropriateness 
were associated with referral. However, in the two other analyses 
low ratings were associated with referral. This was also the case 
for teachers having children and teacher marital status. In one 
analysis the teacher having children was associated with referral; 
in two others not having children was associated with referral. 
Further, in one analysis the teacher being single was predictive of 
referral; in another the teacher being married was predictive of 
referral . 

Table 12 

Discriminant Analy sis Standardized Coefficients 



IS4£_! l£££_2 

aSSiS South Dakota UlSh South Dakota 

Variable Experimental Control Experimental Control Experimental Control EffPfr jme»tSl Control 

Mainstreamed Students .30 .75 

Teacher Ethnicity -.57 
Appropriateness Main- 

streaaino Mild/Moderate .72 -.42 .52 .39 .96 

Appropriateness Main* 

streaming Severe .35 -.78 .38 -.63 .90 

Teacher Gender .81 .77 

Teacher has Children .77 -.42 -.60 

Class Potential .61 .82 .48 

Mifltber of Students in 
Class .66 .46 

Variability Class 

Potential .45 .69 .51 .42 

Teacher Marital Status .30 -.43 



Variability in teachers' perceptions of class potential was 
the variable that most consistently predicted referral in the 
same direction. It appeared as an important predictor in four of 
the eight analyses, and high ratings of variability were 
consistently associated with referral. 
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Implementation of RC in year tvo did not appear to impact 
referral bias in any systematic way. Variables identified as 
important in predicting referral in the experimental groups in 
year one sometimes were and sometimes were not those identified 
as important predictors in year two. Further, the same variables 
were not necessarily identified as important predictors in the 
two experimental groups in either year. 

Accuracy of Classification Decisions 

originally, it was planned that comparisons would be made 
among the classification decisions of 1) the multidisciplinary 
teams placing students in Utah schools, 2) Utah State Office of 
Education (USOE) classification experts, and 3) expert systems 
computer programs designed tc give a second opinion regarding 
classification of students as learning disabled (LD) , seriously 
emotionally disturbed (SED) , mentally retarded (MR) , 
orthopedically impaired (01) and speech impaired (SI) . 
Unfortunately, two factors combined to make the planned analysis 
impossible. First, too few students were identified as MR, 01, 
and SI to do a meaningful analysis. Second, as project staff 
reviewed the files of students classified by the 
multidisciplinary teams as LD and SED, and ran consultations 
using the appropriate expert systems it became apparent that the 
large majority of files contained too little classification data 
to run a meaningful consultation. 

In an attempt to estimate the relationship between 
multidisciplinary team, USOE expert, and expert system 
classification decisions a small sample of files on students 
classified as LD and SED were submitted to USOE experts for 
review. Three experts reviewed each file and gave their opinions 
as to whether the students should be classified as disabled and 
qualified for special education services. Data were also 
available from consultations with the LD expert system for most 
students classified as LD. However, it needs to be stressed that 
the results of these consultations are suspect because project 
staff were often forced to make assumptions and guesses when data 
were not available in the files reviewed. Unfortunately, none of 
the files on the SED students contained enough data to run a 
meaningful expert system consultation. 

Table 13 presents a summary of the LD and SED classification 
decisions made by the multidisciplinary teams, USOE experts, and the 
LD expert system. The multidisciplinary teams classified all of the 
students assessed for learning disabilities as LD. USOE experts 
agreed with these classification decisions in only two cases (20 %) 
and disagreed in eight (80%). The LD expert system conclusions 
agreed with the multidisciplinary team and the USOE experts on one 
case where the latter two agreed the student should be classified as 
LD. In two cases the LD expert system conclusions were borderline 
indicating the classification decision could go either way. of the 
remaining six cases where a decision was available from all three 
entities the LD expert system conclusions agreed with the 
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muitidisciplinary teams in four cases (67%) and with the USOE 
experts in two cases (33%)* The muitidisciplinary teams also 
classified all of the students assessed for serious emotional 
disturbance as SED. USOE expert* agreed with only one of these 
decisions (10%). Ninety percent of the time the USOE experts 
disagree with the muitidisciplinary teams' decisions to classify 
students as SED. 

Table 13 

Comparison s of Muitidisciplinary Team. USOE Expert, and LP expert 
System Classification Decisions 



LP Students Team Decision USOE Decision 



1 
2 
3 



6 
7 
8 
9 
10 

SED Students 

1 
2 
3 
4 
5 
6 
7 
8 
9 
10 



yes 
yes 
yes 

yes 

yes 

yes 
yes 
yes 
yes 
yes 



yes 
yes 
yes 
yes 
yes 
yes 
yes 
yes 
yes 
yes 



no 

no* 

no 

no 

yes 

yes* 

no 

no* 

no* 

no 



no* 

no 

no 

no* 

no* 

yes* 

no 

no 

no* 

no* 



Expert System 
Decision 

no 
no 

Borderline 
Borderline 



yes 
yes 
yes 
yes 
yes 



* USOE experts were divided on these decisions 



Accuracy of RC 

The final planned analysis was to examine the relationship 
between RC referral decisions and the classification decisions of 
expert systems. Unfortunately, failure to obtain sufficient , 
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meaningful, expert system consultation decisions made this 
analysis impossible. 



User Suggestions 

Project staff worked closely with teachers and special 
education supervisors at the schools where RC was implemented 
during year two. As a result of this experience staff had 
frequent opportunities to obtain feedback on RC and suggestions 
for improving its usefulness to teachers. A summary of the 
suggestions most frequently made follows: 

1. RC requires that teachers input information on what 
skills from the Utah Core Curriculum average students in the 
class have and have not mastered. Further, the same information 
is required on the student being considered for special education 
referral. A frequent comment was that something should be done to 
shorten this data gathering process and make things easier for 
the referring teacher. 

2. RC collects data such as that described in item 1 for any 
of four academic areas (math, reading, science, and social 
studies) the refer* * /g teacher indicates are a concern. The 
program assumes that Mm student being referred has mastered at 
least as many skills aj the average student in the class for all 
areas that are not identified as a concern and this information 
is included in the consultation final report. Some teachers felt 
that this is not a safe assumption and that to report based on it 
that a student has mastered 100 percent of the skills the average 
studer.*. in the class has mastered may be misleading. 

3. Most teachers suggested that the Utah Core Curriculum be 
revised. They expressed the opinion that some areas don't apply or 
should be weighted differently in the referral decision process. 

4. RC has an "error loop" that allows a teacher running a 
consultation to go back to any section of the program in which an 
error has been made and correct it. This feature is not available 
for correcting demographic information on the student asked for 
at the beginning of a consultation. Some teachers suggested that 
this feature be incorporated. 

5. Some teachers suggested that RC's final report be printed 
on "NCR" paper so that copies were readily available. 

6. Some teachers suggested that SAT scores and other 
achievement information be considered in the RC referral decision 
process. 

7. RC presents a menu of behavior problems that research has 
shown indicates a student is likely Lo have difficulty in school 
and asks if the student exhibits any of those behaviors. Some 
teachers suggested that this section be expanded to allow the 
teacher to input information on additional problem behaviors not 
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on tiie list 



Discussion 

Associated with the implementation of RC in year two of the 
present study was a tendency for the percent of students referred 
for special education to decrease in both the experimental and 
control groups. This tendency was more marked in the experimental 
than in the control groups, but the effect was not found to be 
statistically significant. similarly, a tendency for lower 
percentages of students to be placed in special education in year 
two was associated with implementation of RC for both the 
experimental and control groups. This effect was not found to be 
statistically significant and no differences between the 
experimental and control groups were observed. Finally, 
implementation of RC had no statiscally significant effect on the 
ratio of students placed in special education to students referred 
for assessment. 

In summary, implementation of RC had no statistically 
significant impact on the percentage of students referred for 
special education assessment, the percentage of students placed in 
special education, or the ratio of students placed to students 
assessed. Trends in the data suggest RC may have some slight effect 
in decreasing the percent of students referred for assessment. At 
best, implementation of RC, and consideration of the factors 
associated with its implementation by teachers and assessment 
personnel in making referrals, may be a necessary but not 
sufficient condition for improving the referral/assessment/placement 
process. 

Coincident with the implementation of RC, there was a highly 
significant decrease from year one to year two in the mean hours of 
assessment time devoted to students in both the experimental and 
control groups. Further, this effect was significantly more 
pronounced in the control than in the experimental groups. During 
year one, significantly more assessment time was devoted to students 
in the control groups (UT - x ■ 1.1.40, SD - x - 28.26) than in the 
experimental groups (UT - x - 8.81, SD - x - 10.23). In year two 
the mean amount of assessment decreased for all groups to be roughly 
equ>l (UTexp. - x - 5.79, UTcon. - x ■ 6.23, SDexp. - x = 7.64, 
SDcon. - x » 6.81)* The reasons for this change are not clear. 

Implementation of RC had no discernible impact on the effects 
of variables biasing the referral process. However, perhaps the 
most interesting finding in the entire study was that variables 
predicting referral for special education assessment were not 
consistent, in the present study, eight discriminate analyses 
identified no variable that consistently predicted referral for 
special education assessment. Further, some variables were found to 
predict being referred in one analysis and not being referred in 
another. These findings suggest that there is no simple 
relationship between the variables studied and whether or not a 
student will be referred for special education assessment. Rather, 
it appears that complicated interactions are at work and that the 
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results of previous studies have painted a less than complete 
picture of how these variables function in predicting referral. 
Much additional research might be directed toward unraveling this 
puzzle. 

The zero to minimal impact implementation of RC had in 
improving the referral/assessment/placement process was, of course, 
disappointing to the project staff. Never-the-less, staff are 
committed to developing expert systems that will assist special 
educators in improving that process. In the future, RC will be 
revised to incorporate those changes users in the present study 
suggested. Further, project staff will attempt to find new ways to 
implement the system and continue to assess its impact on the 
referral/ assessment/placement process . 
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Overview 

1. Referral Consultant (RC) first asks a series of questions to 
determine if a student should obviously be referred for 
special education assessment. If it concludes the student 
should obviously be referred it sends a message to do so. 

2. For s'.udents who ire not obvious referrals RC first compares 
what the target student can do academically to what the 
average student in his/her class can do. This information 
is used to arrive at a confidence that the student has 
learning problems or adverse effects on educational 
performance. The confidence is expressed as a number from 1 
to 100 and is similar to a percent. For example, RC might 
arrive at a confidence factor (cf) of 80, indicating it is 



80 percent confident that the student is experiencing 
learning problems. Unless information provided on the home 
environment, culture, and/or educational history indicates 
otherwise this is also RC's confidence that the student 
should be referred for special education assessment. 

After determining a student is experiencing learning 
problems, RC goes on to ask questions that lead it to 
conclusions regarding whether those learning problems can be 
attributed to 1) behavioral/social-emotional, 2) sensory 
(vision or hearing) , or 3) physical/health difficulties. 
Separate confidence factors indicating how confident RC is 
that learning problems are due to behavioral or physical 
difficulties are assigned. For example, RC might conclude 
that a student's learning problems could be accounted for by 
behavioral/social-emotional difficulties at a confidence 
factor of 95. This would indicate RC is 95 percent 
confident learning problems were due to behaviorai/social- 
- emotional difficulties. Observations suggesting the student 
may have vision of hearing problems are also noted. 

Next, RC asks questions that lead it to conclusions 
regarding whether a student's learning problems can be 
accounted for by non-special education difficulties. The 
three areas considered are the home environment, cultural 
differences, and educational history. Separate confidence 
factors indicate how confident RC is that learning problems 
are due to difficulties in these areas. 

Finally, RC asks if two "quality" pre-referral interventions 
have been tried to help the student. If this has not been 
done, RC sends a message that its other conclusions are 
tentative and that appropriate pre-referral interventions 
should be tried. 
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Referral Consultant L. ic 



Obvious Referral 



System Questions and Rules 

RC asks the following question to get at whether the student 
should obviously be referred for special education assessment: 

Does the student exhibit any of the following conditions which 
would obviously be grounds for referral for special education 
assessment? Enter the number of each condition that applies 
separating them with commas (e.g. 2,3,5). The student: 

1. exhibits NONE of the below. 

2. has LITTLE or NO ability to speak or understand 
LANGUAGE . 

3. exhibits extreme PSYCHOTIC or obviously BIZARRE BEHAVIOR 
that interferes with educational performance. 

4. is obviously BLIND or DEAF. 

5. is so shy and withdrawn that he/she never or RARELY 
INTERACTS with others, and educational performance is impaired 

6. is OBVIOUSLY FAR BELOW his/her peers in ACADEMIC 
ACHIEVEMENT. 

7. has a SEVERE PHYSICAL/HEALTH IMPAIRMENT that interferes 
with educational performance. 

8. exhibits an obvious reason for referral which is not 
listed. 

If the user responds with #1, RC moves on continuing the 
consultation. 

If the user responds with #8, RC asks the user to input the other 
reasons: 

Please enter the obvious reason for referral that applies to 
this student which wasn*t on the list. The student: 

If the user responds with #2-8, RC sends a message that the 
student should be referred that lists all the reasons selected 
and ends the consultation. 



3 

41 



Referral Consultant Logic 



Pre-referral Intervention 



Referral Consultant asks what pre-referral interventions 
have been tried with the student. If attempts to remediate the 
student's problem (s) through non-special education interventions 
were not tried, there may be no need to go all the way through a 
consultation. If two quality interventions have not been tried, 
the user is given the option to quit or continue. The question 
asked is this: 

TWO or more QUALITY INTERVENTIONS must be attempted by regular 
educators prior to referring a student for special education 
assessment . 

Examples of acceptable quality interventions include: 
behavior management programs, a change of classroom teacher, 
adjusting academic variables (e.g. curriculum, materials, 
schedule) , chapter I or other remedial programming, peer 
tutoring, etc. 

Examples of non-acceptable interventions include 
parent/child conferences that do not result in specific actions 
to be taken. 

NOTE: For an explanation of what constitutes QUALITY 
interventions type "why". 

Have two or more quality interventions been attempted by 
regular educators? 

1. yes 

2. no 

If the user types "why": 

Quality interventions are those which meet the following 
criteria: 

- The child and his/her parents were informed of the problem. 

- Plans were developed to remediate the problem. 

- A team of educators feels that the plans were appropriate 
and well implemented. 

- Documentation through data collection on the interventions' 
effectiveness was obtained for at least one week. 

If the user answers "yes", the program asks the user to type in 
the interventions tried and the following message is incorporated 
into the final report: 
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The following pre-referral interventions have been tried with 
this student: 

(List input by user) 

If the user answers "no", he/she is given the message below and 
asked if he/she would like to continue the consultation. If the 
user chooses to continue a similar message is incorporated into 
the final report. 

Best practices in special education dictate that before a 
student is referred for special education assessment at least 
two quality interventions be tried to correct the student's 
learning problems. Since this has not been done conclusions 
drawn by Referral Consultant at the end of this consultation 
will be tentative. It is recommended that eppropriate 
interventions be attempted and that a new Referral Consultant 
consultation be run based on the student's status following 
those interventions. 

Do you wish to continue the consultation? 

1. yes 

2 . no 
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Referral Consultant 



Learning Problems 
(Adverse Effects on Educational Performance) 



System Que stions and Rules 

RC asks the following question to determine where the 
student is having learning problems. 

In which of the following areas is the student experiencing 
learning problems? Enter the number of each area separating 
them with commas (e.g. 2,3) . 

1. None of the below. 

2. Reading/Language Arts 

3 . Mathematics 

4 . Science 

5. Social Studies 

If #1 is chosen, the following statement appears: 

The student is not experiencing learning problems in any 
major academic area and, therefore, should not be referred for 
special education assessment. However, continuing the 
consultation may help you determine if the student should be 
referred for non-special education services. Do you wish to 
continue? 

1. yes 

2. no 

If any combination of #2-5 is chosen: 

For each area indicated, RC next asks what skills the 
average student of the same sex in the class has. The skills are 
taken from the Utah Core Curriculum. 

The skills are presented on the computer screen in menus. 
For example: 

From the following Utah Core curriculum objectives, which of 
the following skills CAN the AVERAGE male student in Dick's 
class do? 

1. None of the below 

2. Identify, read, and write any given numeral to 10,000. 

3. Recognize that multiple digit numerals are grouped 
into periods of threa digits. 

4. Identify the place value of a digit in numerals to 10,000. 
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5. Demonstrate place value to 9,999 using expanded notation, e.g., 
9,999 -9,000 + 900 + 90 + 9 ■ x9 thousands + 9 hundreds + 

9 tsns + 9 ones. 

6. Use halves, thirds, or fourths to identify the fractional 
parts of a set of region. 

7. Compare any two numbers between 1 and 9,999 using 
symbols greater than (>) , less .than (<) , or equal («) . 

There are between 2 and 5 menus for each subject per grade. 
The first group of menus presents skills from the target 
student's grade level. If the average student can do less than 
half the skills at grade level, RC drops back a grade. A cycle 
going down continues until more than half the objectives are 
chosen or grade 0 is reached. If the average student can do 
more than half the objectives from the beginning grade level, RC 
moves up a grade level. This cycle continues up until the 
average student can do less than half of the objectives, or until 
gra.de 12 is reached. Thus, skills over at least two grade 
levels are sampled. 

Example: Dick is in 2nd grade and has trouble in 
mathematics. Mr. Brown, the concerned teacher, is presented 
the menus for 2nd grade math. Mr. Brown inputs the skills 
which the average student in his class can do. The average 
student in Mr. Brown's class can do more than half of the 
skills from the 2nd grade math menus, therefore, the program 
moves to the third grade menus. Mr. Brown inputs what 
skills the average student in his class can do from these 
menus. It is less than half and therefore RC would move 
down a grade, but since the 2nd grade menus have already 
been shown, the cycle quits. 

After determining the skills the average student can do, RC 
combines them into menus and asks which of the skills the target 
student cannot do. For example; 

You have identified the following list of skills which the 
AVERAGE STUDENT in Dick's class CAN DO. Which can Dick NOT do? 

1. Dick can do All of the below. 

2. Identify, read, and write any given numeral to 1,000. 

3. Identify the place value of a digit by its position in 
the ones, tens, or hundreds ' place. 

4. Demonstrate place value to 999 using expanded notation, e.g. 
999 - 900 +90+9; 999 - 9 hundreds + 9 tens + 9 ones. 

5. Identify and read the symbols 1/2, 1/3, 1/4 as names 
for some fractions. 

6. Count and order numbers by l % s to 1,000 & 2*s to 100. 

Using the information provided RC calculates what percent of 
skills the target student has in each area, i.e., math, reading, 
science, and social studies. For example, if the average student 
has 50 math skills and the target student has 10, the target 
student has 20 percent of the math skills the average student 
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has. If the user indicates the student was not having trouble in 
an area, RC assumes the target student has 100 percent of the 
skills the average student has. 

RC applies two rules to the four percents of skills to 
arrive at a conclusion regarding whether the student is having 
substantial learning problems. First, it considers the 
discrepancy between the lowest of the math or reading percents 
and the highest other percent. For example, if a student's 
percents were math ■ 20, reading * 80, science, m 85, social 
studies - 80, RC considers the discrepancy between science (85 is 
the high score) and math (20 is the lowest of reading and math) . 
The discrepancy is calculated as science (85) - math (20) « 65. 
RC is 65 percent certain the student has learning problems. The 
rationale for this decision is that a student who is potentially 
L0 will show a large discrepancy between an area where he/she 
could qualify as LD (math or reading) and other areas of 
academic achievement. Finally, the following regression equation 
describes the relationship between the discrepancy and the 
confidence RC has that a student has learning problems. 



100 

Discrepancy 




0 100 
Confidence Student 
Has Learning Problems 

The other rule RC applies to the four percents is to average 
them. For example, if a student's percents were math ■ 20, 
reading ■ 25, science - 15, and social studies ■ 20, the average 
would be 20. This average figure is entered into the following 
regression equation to arrive at a confidence the student has 
learning problems. 



90 
80 

Average 70 
% of Skills 

40 




0 5 10 25 99 
Confidence student 
Has Learning Problems 

A student with an average score of 40 has 40 percent of the 
skills the average student in the class has. Since the percent 
of skills is low th*! regression equation leads us to high 
confidence 99% that the student has learning problems. The 
rational is that an MR student will have low percents across all 
four areas and therefore a low average. 
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rc takes the highest of the two confidences generated by 
applying the rules as the best estimate that the student has 
learning problems. 

If RC's confidence that a student has learning problems is less 
than 20 percent the following message is displayed to the screen. 

There is less than a 20% chance that Dick is experiencing 
learning problems that would require special education services 
(Certainty Factor ■ CF, '/100) . 

Referral Consultant will next present a series of questions 
to examine whether Dick could benefit from non-special 
education services such as counseling, a multi-cultural 
program, etc. Do you wish to continue the consultation? 

1. Yes, X wish to continue 

2. No, end the consultation 
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Referral Consultant 
Behavior 



Granite school district has coma with an innovative idea for 
deciding if learning problems are the result of behavior 
problems. It is based on the work of Hill Walker and his 
colleagues at the University of Oregon who have been developing a 
system for screening behaviorally disordered students. Walker 
et.al. have collected data indicating that certain critical 
events (behaviors) discriminate students who are at high risk for 
behavior disorders. Those as listed on their screening 
instrument events are: 



SSBD Stage Two For Externalizing student 
Critical Events Index 

Date Teacher School 

Student Student Sex Student Grade 

Check one: Rank 1 2 or 3 

instructions: Check each behavior that you are aware the student 
has exhibited during this school year. Please specify any 
serious behavior not appearing on this list. 

1. Steals. 



2. Sets fires. 



3. Vomits after eating. 

4. Has tantrums. 



5. Physically assaults an adult. 

6. Exhibits painful shyness. 

/. Exhibits large weight loss or gain over past three 
months. (Significant weight fluctuation would be in 
excess of 20% change in body weight.) 

8. Exhibits sad affect, depression and feelings of 
worthlessness to such an extent as to interfere with 
normal peer and classroom activities. 
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9. Is physically aggressive with other students or adults 
(hits, bites, chokes, or throws things. 

10. Damages others' property (academic materials, damages 
personal possessions) . 

11. Demonstrates obsessive-compulsive behaviors. (Child 
can't get his/her mind off certain thoughts or 
obsessions) . 

12. Reports having nightmares or significant sleep 
disturbances . 

13. Engages in inappropriate sexual behaviors 
(masturbation, exposes self) . 

14. Is self-abusive (biting, cutting or bruising self, 
head banging) • 

15. Attempts to seriously physically injure another using 
weapons or objects. 

16. Suddenly cries or displays highly inappropriate affect 
in normal situations. 

17. Complains of severe headaches or other somatic 
complaints such as stomach aches, nausea, dizziness, 
or vomiting. 

18. Talks of killing him\herself. Reports having suicidal 
thoughts or being preoccupied with death. 

19. Exhibits thought disorders or gets lost in own 
thoughts . 

20. ignores teacher warnings or reprimands. 

21. Makes lewd or obscene gestures. 

22. Shows evidence of physical abuse. 

23. Shows evidence of drug use. 

24. Reports being sexually abused. 

25. Uses obscene language or swears. 

26. Exhibits cruelty to animals. 

27. Is teased, neglected and/or avoided by peers. 

28. Has severely restricted activity levels. 

29. Is enuretic (bed wetting). 
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30. is ancopretic (inadequate bowel control)* 

31. Sexually molests other children. 

32. Has auditory or visual hallucinations. 

33. Has severe lack of interest in activities which were 
previously of interest. 
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walker et.al. have collected data indicating that students 
identified by teachers as fitting profiles of externalizing and 
internalizing behavior disorders score higher on the Critical 
Events Scale than normal control students. Briefly, 
externalizers are students who act out, and internal izers are 
students who are shy and withdrawn. The following table presents 
means and standard deviations for various groups on the Critical 
Events Scale. 



TABLE 1 

Means and Standard Deviations by Group and Sex 
for Items Checked on the SSBD critical Events Index 





X 


SD 


N 


Externalizers 








All 


5. 27 


3 .21 


33 


Male 


5.69 


3. 10 


29 


Female 


2.25 


1.69 


4 


Internal izers 








All 


3.73 


2.83 


15 


Male 


2.71 


1.58 


7 


Female 


4.63 


2.49 


8 


Controls 








All 


.i; 


.43 


235 


Male 


.12 


.45 


119 


Female 


.11 


.41 


116 



RC presents the critical events and ask which of the 
behaviors the student has exhibited in the current school year. 
The number, of behaviors engaged in is entered into one of the 
following equations to assign a probability that learning 
problems result from behavior problems. 
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Females 

10 

# Critical 
Events 5 



0 50 
Confidence Learning 
Result 



100 
Problems 
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Referral Consultant 
Sensory (Vision/Hearing) 

v to ion 

In dealing with vision RC asks the following question: 

Does the student exhibit any of the following symptoms of 
possible visual impairment? Separate multiple answers with 
commas (e.g. 2,5,6). 



1. 


NONE ox tne below. 


16. 


Fails to see oojects 


2. 


Rubs eyes 




in peripheral vision 


3. 


Headaches 


17. 


Holds written work close 


4. 


Slow reader 




or far away to see it 


5. 


Frequent sties 


18. 


Difficulty reading the 


6. 


Bloodshot eyes 




chalkboard (squints) 


7. 


Oversensitive to light 


19. 


Difficulty reading written 


8. 


Tilts head to read 




work (holds away) 


9. 


Eyes in constant motion 


20. 


Crossed eyes 


10. 


Double vision 


21. 


Swollen eyelids 


11. 


Frequent pain in eyes 


22. 


Eyes not functioning together 


12. 


Unable to distinguish color 


23. 


Red rimmed or crusted eyelids 


13. 


Shuts or covers one 


24. 


Blinks constantly 




eye when reading 


25. 


Pupils of different sizes 


14. 


Confuses similar letters 


26. 


Frequently trips or stumbles 




(e.g. m and n) 


27. 


Walks with extreme caution 


15. 


Frowns or squints when 


2b, 


Burning or itching eyelids 




trying to see objects 


29. 


Complains of blurred vision 






30. 


Otusr 



If other is chosen RC reads the following message: 

Please enter the other symptoms of possible visual 
impairment you have observed. Separate multiple answers 
with commas. 



If none is chosen RC skips to the next section of the 
consultation. 

If any combination of 2-30 is chosen RC notice all symptoms and 
presents them in the final report. 



o 

ERIC 



15 

53 



Hearing 



In dealing with hearing RC begins with the following 
question. 

Does Dick exhibit any of the following symptoms of possible 
HEARING impairment? 

(Enter the number of all that apply, separating them with 
commas, e.g. 2,5,6.) 

1. NONE of the below. 

2. seems not to hear or misinterprets instructions, and may 
be frustrated 

3. rubs or pulls at ears 

4. Asks others to repeat a lot (what, huh) 

5. responds to questions with out-of -context remarks 

6. continual earaches 

7. slurred speech 

8. incorrect pronunciation/misunderstood 

9. difficulty comprehending material read orally 

10. speaks with a loud voice 

11. speaks with a flat or monotone voice 

12. daydreams and doesn't seem to pay attention 

13. difficulty understanding when speaker is behind or 
across the room from Dick. 

14 . Other 

If other is chosen RC sends the following message: 

Please enter the other symptoms of possible hearing 
impairment you have observed. Separate multiple answers. 

If none is chosen RC skips to the next section of the 
consultation. 

If any combination of 2-14 is chosen RC notes all symptoms and 
presents them in the final report. 
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Referral Consultant 
Physical/Health 



system Qu estions and Rules 

RC next attempts to determine if physical/health problems 
can account for the student's learning problems. It first asks 
if the student has any of the following diagnosed conditions: 



Has the student been diagnosed by a physician to have any of 
the following problems? Separate multiple answers with commas 
(e.g. , 2,5, 10) . 



1 


S3 


NONE of the below 


12 


SB 


leukemia 


2 




amputation 


11 




mental retardation 


3 




arthogryposis (persistent 


14 


S3 


motor dysfunction 






flexture or contracture 


15 


s 


muscular dystrophy 






of a joint) 


16 




nephritis 


4 




asthma 






(kidney inflammation) 


5 


n 


brain injury 


17 


S3 


respiratory disorder 


6 


S3 


cerebral palsy 


18 




rheumatic fever 


7 




diabetes 


19 


ES 


sickle cell anemia 


8 




epilepsy 


20 


S* 


spina bifida 


9 




heart condition 


21 




spinal cord injury 


10 


BS 


hemophilia 


22 




tuberculosis 


11 


rs 


lead poisoning 


23 




other 



If other is chosen RC asks: 

What other diagnosed problems does the student have? 
(Separate multiple answers with commas.) 

RC then asks if the student exhibits any of the following 
symptoms : 

Does the student FREQUENTLY exhibit any of the following 
symptoms? Separate multiple answers with commas. 

1. None of the below. 

2 . Tiredness 

3 . Listlessness 

4. Complains of pain 

5. Absent for illness 

6. Petite mal seizures, brief lapses of consciousness 

7. Runny nose 

9. Chronic symptoms (headaches, stomachaches) 

10 . Vomiting 

11 . Diahrrea 

12. Complains of feeling ill 
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13 . Thirsty 

14. urination 

15. Other 

If other is chosen RC asks: 

What other frequent symptoms does the student exhibit? 
(Separate multiple answers with commas.) 

Finally RC asks the user to estimate on a scale of 0 - 100 
how much the diagnosed problems or symptoms account for learning 
problems. This is taken as the confidence that physical/health 
problems account for learning problems. 

How much do the student *s physical/health problems and 
symptoms (LIST OF PROBLEMS & SYMPTOMS) contribute to adverse 
effects on educational performance? 

, 1 1 1 1 1 ,„.| j— -| 1 

0 50 100 

Not at all About Half All 



RC also collects information on medications taken by the 
student. 

Is this student taking ANY medications regularly? 

1. yes 

2. no 

If yes is chosen RC asks: 

Why does the student take medications regularly? Input the 
number (s) of all that apply separating them with commas. 

1. control of behavioral problems 

2. control of physical/health problems 

3. unknown reasons 

If behavioral (#1) - What medications does the student 
take regularly to control behavioral problems? 
Separate multiple answers with commas. 

If physical (#2) - What medications does the student 
take regularly to control physical/health problems? 
Separate multiple answers with commas. 

If unknown (#3) - What medications does the student 
take regularly for reasons unknown to you? Separate 
multiple answers with commas. 

The following message is incorporated in the final report. 

Based on the information provided you can be percent 

certain that the student % s learning problems may be due to 
physical problems. 
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The following diagnosed physical/health problem (s) were 
noted: 

(list of physical problems) 

The following frequent symptoms were noted: 

(list of symptoms) 
The student is taking the following medications 
regularly: 

For behavioral control: 

(list) 

For control of physical/health problems: 

(list) 

For unknown reasons: 

(list) 
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Referral Consultant 

Home Environment 

RC next attempts to determine if problems in the home 
environment can account for the student's academic problems. It 
asks if any documented problems such as divorce or death of a 
family member exist. For each problem noted it assigns a 
confidence factor of 20. Using special combination rules RC 
combines the cf 20s for all home problems noted into an oyerall 
confidence that learning problems result from problems in the 
home environment. 
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Culture 



Questions and System Rules 

RC considers teacher ratings of English proficiency, years 
of English as a second language instruction (ESL) , student ethnic 
background, the percent of minority students in the school, and 
years as a resident of the US in arriving at a confidence that 
learning problems result from cultural differences. 

Questions presented and rules tor considering the 
information provided are as follows. 

1. In comparison to chronological age peers how would you 
rate the students' English proficiency? 

1. Better than Average (cf -15) 

2. Average (cf 0) 

3. Less than Average (cf 15) 

If "Less than Average 1 * (#3) is chosen: 

A. Is the primary language used in Dick's non-school 
environment something other than English? 

1. Yes 

2. No 

If "Yes" (#1) is chosen: 

B. How many years of English as a second language 
instruction has the student had? 



5 K 

ESL I \ 



Years 



0 30 100 
Confidence Cultural Effects Learning 



2. What is the student's ethnic background? 
Native American (cf-10) 
Hispanic (cf-6) 
Asian/Pacific Islands (cf-3) 
Other (cf-3) 



21 

59 



3. What percent of students in the school belong to the sane 
minority? 

100 

Percent 
Minority 

30 
10 

0 7.0 15 22 100 
Confidence Culture Effects Learning 




4. How many years has the student been a resident of the 
US? 



5. What is the student's current grade placement? 



Percent 
Education 
Out of US 



100 



50 



if years in US <= 
grade placement 



0 20 40 100 
Confidence Culture 
Effects Learning 



Percent 
Life Out 
of US 



100 

60 
50 



if years in US > 
grade placement 



8 10 100 
Confidence Culture 
Effects Learning 



RC uses special rules to combine the individual confidence 
factors generated by the rules into an overall confidence that 
cultural factors are causing learning difficulties. 
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Referral Consultant 
Educational History 



Questions and System Rules 

RC asks the following questions to determine if certain 
educational history factors might account for academic problems. 



Has Dick experienced any of the following in his educational 
history. Please enter all that apply. 

The student: 

1. NONE of the below. 

2. has changed schools in the past year or two. (cf 10) 

3. has changed schools frequently in academic career, (cf 20) 

4. had poor attendance in the past, (cf 20) 

5. has experienced a major change in instructional strategy 

(e.g., from phonic to whole language approach to reading), 
(cf 20) 

6. Other educational history problem. 

RC combines the confidence factors shown in parentheses 
above to arrive at an overall confidence that educational history 
problems account for the student's learning problems. 

If # 8 is chosen, the user is asked to type in the other 
educational history problem(s). Then a scale is presented on 
which the user is to rate the impact of "other" problems on the 
student % s learning problems. 

The final report states: 

Based on the information provided you can be percent 

certain that the student has learning problems that result 
from an unusual educational history. 

Students whose learning problems are primarily due to an 
unusual educational history may benefit from remedial instruction 
or other non-special education services offered by the school 
dirtrict. 

Specific educational history problems noted include: 
The student: 
(list) 
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Referral Consultant 
Final Report 



After collecting the information described in the previous 
sections RC develops a report summarizing the data and its 
conclusions regarding whether the student should be referred. 
Sample reports are containing in the Appendix. The report is 
divided into five sections. 

1. Confidence That Student Has Learning Problem? 

This section presents RCs confidence that the student is 
experiencing learning difficulties. It is also RCs confidence 
that the student should be referred for special education 
assessment or other services. If the user has indicated that 
there is no reason to believe the student is being Impacted by 
non-special education factors (culture, home environment, or 
educational history) the RC report indicates that the confidence 
presented the confidence that the student should be referred for 
special education assessment. For example: 

CONFIDENCE THAT STUDENT HAS LEARNING PROBLEMS 

Based on the information provided you can be 79.0 percent 
certain that Joe has learning problems. 

You can also be 79.0 percent certain Joe should be referred 
for special education assessment. 

If the user has indicated that the student is being impacted by 
non-special education factors the RC report indicates which 
factors, it also recommends that information provided in a later 
section of the report on non-special education causes of learning 
problems be used to estimate how much confidence can be had that 
the student should be referred for special education assessment. 
For example: 

CONFIDENCE THAT STUDENT HAS LEARNING PROBLEMS 

Based on the information provided you can be 90.0 percent 
certain that Dick has learning problems. 

Information provided in the section below titled "Possible Non- 
Special Education Related Causes of Learning Problems" indicates 
that home environment, culture, and educational history problems 
may be contributing to Dick's learning problems. This 
information should be used to estimate how much less than 90.0 
percent certain you are that Dick should be referred for special 
education assessment. 
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Finally, this section of the report lists, reletive to the 
average student, the percents of skills in math, reading, 
science, and social studies that have been mastered by the target 
student . 

2. Possible Special Education Related Causes Of Learning 
Problems 

This section of the report presents the confidence that can 
be had that the student's learning problems may result from 
behavioral/emotional or physical/health difficulties. Specific 
problems in these two areas are noted. Also, any medications the 
student is known to be taking are listed, and any symptoms of 
possible visual or hearing difficulties arc listed. 

3. Possible Non-Special Education Related Causes Of Learning 
Pr oblem s 

This section of the report lists the confidence that can be 
had that cultural, home environment or educational history 
factors are impacting learning. Specific problems in these three 
areas are noted. Since they are non-special education related 
causes of learning problems the information provided is helpful 
in estimating how much learning problems may be accounted for by 
non-special education factors. For example, consider a student 
that RC is 90 percent confident he has learning problems. If 
none of the learning problems result from non-special education 
factors RC is also 90 percent certain he should be referred for 
special education assessment. However, suppose that RC also 
concludes it is 80 percent confident the learning problems result 
from cultural factors. In this case, most of the learning 
problems are accounted for by a non-special education factor and 
the student would not be referred for special education 
assessment. Rather, a more appropriate referral might be to a 
multi-cultural program. In this section of the report, when 
appropriate, RC suggests alternatives to referral for special 
education assessment. 

4. Pre-Referral Interventions 

In this section of the report pre-referral interventions 
attempted to help the student are listed. If none have been 
tried a statement that without such interventions RC's 
conclusions are tentative is presented. Also, the suggestion 
that at least two be tried and another consultation run based on 
the student's status after appropriate pre-referral interventions 
is presented. 

5. Specific Skills Mastered and Not Mastered 

In this section of the report the specific skills the target 
student has not mastered that the average student in his/her 
class has mastered are presented. In addition, the skills the 
target student has mastered that the average student has mastered 
are presented. These are listed for all areas where the student 
is experiencing learning difficulties; mathematics, reading, 
science, and/or social studies. 
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REFERRAL CONSULTANT REPORT 
CONCLUSIONS 



Consultation Date: 
Student's Nane: 
School : 
Grade : 
Age: 

Teacher's Nane: 



1/15/91 

Dick Beer 

USU Elementary 

3.5 

9 years 5 nonths 
Toni Casdorph 



CONFIDENCE THAT STUDENT HAS LEARNING PROBLEMS 

Based on the information provided you can be 73 percent 
certain that Dick has learning problems. 

Information provided in the section below titled "Possible 
Non-Special Education Related Causes of Learning Problems" indicates 
that educational history problems may be contributing to Dick's learning 
problems. This information should be used to estimate how much less than 
73 percent certain you are that Dick should be referred for special 
education assessment. 

Compared to the average student in his class Dick has mastered: 

100 percent of MATHEMATICS skills 

27 percent of READING skills 

100 percent of SCIENCE skills 

100 percent of SOCIAL STUDIES skills 
*********************************** 

POSSIBLE SPECIAL EDUCATION RELATED 
CAUSES OF LEARNING PROBLEMS 



BEHAVIOR/SOCIAL-EMOTIONAL : 

Based on the information provided you can be 
0 percent certain that Dick's learning problems say be 
due to a behavior disorder/serious-emotional disturbance 

PHYSICAL/HEALTH: 



Based on the information provided you can be 
20 percent certain that Dick's learning problems may be 
due to physical problems. 

The following diagnosed physical/health problem(s) 
were noted: 

hyperactivity 

Dick is taking the following medications regularly: 
For behavioral control: 
ritalin 

VISION: 5ft ~~ 
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0 No symptoms of vision problems have been noted. 
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HEARING : 



Mo symptoms of hearing problems have boon notod. 
*********************************** 

POSSIBLE NON-SPECIAL EDUCATION RELATED 
CAUSES OF LEARNING PROBLEMS 

Based on the information provided you can be: 

HOME ENVIRONMENT: 



0 percent certain that Dick has learning problems that 
result froa problens in the home environment. 

CULTURE: 



0 percent certain that Dick has learning problems that 
result from cultural differences. 

EDUCATIONAL HISTORY : 



10 percent certain that Dick has learning problems that 
result from an unusual educational history. 

Students whose learning problems are primarily due to an 
unusual educational history may benefit from remedial 
instruction or other non-special education services. 

Educational history problems noted are listed. Dick: 
CHANGED SCHOOLS in the PAST YEAR or two. 



*********************************** 

P RE-REFERRAL INTERVENTIONS 

The following pre-referral interventions have been tried 
with Dick: 

after school remedial instruction 
peer tutoring 

*********************************** 



Specific Reading/Language Arts Skills Mastered and Not Mastered 



Grade 3 Skills 



Dick HAS MASTERED 
his class CAN do: 
1. Pay attention 
presenting. 
Follow three- 
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the following skills which the average male student 
to the teacher or to others who are speaking or 



2. Follow three- and four-step directions correctly. 

3. Tell major points or sequence of events. 

4. Respond to speakers, e.g.* ask questions and make contributions. 

5. React to literary selections read aloud. 

6. Reoite third grade selections olearly and fluently. 

7. Express and support personal opinions about topics presented. 

8. Respond to opinions expressed by others. 

9. Explain how to do something or tell about an event. 
10. Answer questions accurately. 59 n 
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Ai. ami appropriate questions wnen additional inroraatlon is nasdad. 

12. Build flaonoy in phonatic and structural analysis skills. 

13. Identify the meaning of affixes and root (base) words as they occur in 
the reading task. 

Dick has NOT nastered the following skills which the average sals 
student in his class CAN do: 

1. Know the correct aeaning of common hononyas in context. 

Attack multisyllable words systenatically, e.g., prefix, root word, 
ending . 

Road the sight words and basal vocabulary as they appear in the 
reading program. 

Comprehend word, sentence, and paragraph meanings in context. 
Recognize aain ideas in a selection. 
Alphabetize to the third letter. 
Read and follow directions. 
Read a variety of self -selected aaterials. 
Retell storylines (plots) in the selections. 
Predict logical conclusions to events in the selection. 
Compare characters, events, plots, and settings. 
Recognize cause and effect relationships. 
Utilize major spelling generalizations, e.g., same vowel 
sound/different spellings. 

Spell a basic word list as adopted by the achool. 

Write words and sentences with correct punctuation and capitalization. 
Discriminate between correct and incorrect spelling of words on level. 
Spell hononyas and contractions correctly. 

Show understanding of spelling words by telling what they mean or 
using them in a sentence. 

Use correct formation of all upper and lower case letters and numbers 
in cursive. 

Use proper strokes to join letters to fora words. 
Proof and correct their own handwriting. 
Demonstrate neatness in written work. 
Generate and organize ideas for writing. 
Write personal experiences, stories, poetry, etc. 
Write letters and informative selections. 
Share and respond to the writing of others. 
Use capital letters and terminal punctuation. 
Identify nouns and verbs. 

Participate in a group iaproviaation of a story. 
Stay in character in a short play or skit. 
Speak expressively in a choral or storytelling situation. 
Describe the feelings portrayed in a given picture or situation. 
Identify the setting, plot, and characters in a simple play or story 
Demonstrate and discuss appropriate behavior when viewing a 
performance. 

Give personal reactions after viewing a performance. 
Hake and/or use simple props or costumes to help portray a character. 
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4 
5 
8 
7 
8 
9 
10 
11 
12 
13 

14 
15 
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38 
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Referral Consultant 
Easy 

Reference 

Guide 



Richard Baer, Marilyn Likins, Toni Casdorph, 
Brad Althouse and Joseph Ferrara 



Technology Division 
Developmental Center for Handicapped Persons 
Utah State University 
Logan, Utah, 1989 
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Referral Consultant 



Welcome to Referral Consultant (RC) . This program is 
designed to help you decide if a student should be referred for 
special education assessment, or other non-special education 
services. RC will ask you questions pertaining to a student's 
academic skills, behavior, sensory ability (vision and hearing) , 
health, home environment, culture, and educational history. It 
will then use the information to develop a report stating how 
confident you can be that the student should be referred. The 
child study team may use this report as an aid in the 
referral/assessment process. 

To run Referral Consultant, you must have an IBM, or an IBM 
compatible computer with a minimum of 640K memory and two floppy 
disk drives. A printer is not required, but may be used at the 
end of the consultation to provide the user with a print-out of 
the consultation and/ or final report. 



DATA DISKS; 

Before beginning a consultation make sure you have a 
formatted computer disk. Thxs disk will be used to store a 
record of the consultation and report for the student being 
consulted about. To format a new disk use the FORMAT command in 
the DOS section of this manual. 

getting STARTED: 

There are two ways to start referral consultant: 

1. If your eomputer is off: place the RC program disk 
labeled "Disk A" in the A drive and the disk labeled 
"Disk B" in the B drive. If your computer disk drives 
are sida-by-side, the A drive is on the left. If your 
computer disk drives are on top of one another, the A 
drive is on the top. 

Turn the machine on and the computer will automatically 
load the program. 

2. If your eomputer is on and an A> is on the screen: 

place the RC program disks in the A and B drives and 
type RC, then hit the enter/ return key. 



Conducting & Consultation: 

After the RC program is loaded, the system will begin to ask 
you questions about the student. Enter the requested information 
and then strike the enter/return key. 
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CORRECTING MISTAKES: 



There are two ways to correct a mistake: 

1. If you have made an error hut have not yet pressed the 
enter/return key: simply use the arrow or backspace key 
to back up and then re-enter your answer. 

2. If you have made an error and pressed the enter/rsturn 
key: a menu near the end of the consultation will allow 
you to correct mistakes in any or all of the main 
sections of the program. However, demographic 
information gathered at the beginning of the program 
(name, date, grade, etc.) cannot be changed. If a 
mistake is make in entering demographic information the 
consultation should be restarted. 



PROGRAM COMMANDS: 

WHY: Typing why in response to a question will 1) bring up an 
explanation of why the question is being asked, and 2) then re- 
ask the question. For example: 

Please enter the student's FIRST name. 
»why 

The student's name, today's date, and the name of the school 
are needed to make a complete record for the consultation. 

Please enter the student's FIRST name. 
»Dick 



UNKNOWN: If you don't know the answer to a question, type the 
word unknown and press the enter/ return key. RC accepts this 
answer, but, lowers the confidence of its conclusions. For 
example: 

Which school does Dick Baer currently attend? 
>> unknown 



BACKSLASH \: If your answer will not fit on one line, type a 
backslash \ at the end of a line, then hit the enter/return key 
to move to the next line. For example: 

Enter the quality interventions tried with Dick. 
Type a comma between each anger. 

» behavior management programs, a change of classroom\ 
» teacher, adjusting curriculum 

The backslash allows you to extend your answer to the next line. 
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D1TXFILE8 

Two types of files are automatically created during a 
consultation. 

1. Report example: smiraywo . rpt 

2. Consultation example: smiraywo.log 



A filename consists of 8 letters plus type of file: 

-the first 3 letters of a student's last name, 
-the first 3 letters of a student's first name, 
-and the first 2 letters of the school's name. 

A filename for Ray Smith, a student at Woodruff Elementary would 
be smiraywo. rpt. 



BOS APPLICATIONS: 

Drive Commands 

Command 

A>b: (press return) 
B>a: (press return) 
c>a: (press return) 



Function 

Changes disk drive from A to B 
Changes disk drive from B to A 
Changes disk drive from C (hard 
disk) to A. 



Directory Cpmmands 

Command F unction 

A> dir (press return) Calls up the directory for the A 

drive and lists all files. 

B> dir (press return) Calls up the directory for the B 

drive and lists all files. 



Command 

A> format b: (press return) 
c> format a: (press return) 



Function 

formats a blank disk in B 
drive. 

formats a blank disk in A 
drive. 



PROBLEM ALERT 
When formatting a disk with a hard disk system, be sure 
to type a: after the word format. If you forget to type 
as after the word format, you may format your hard disk 
drive which erases all files on your hard disk. This is 
a BIO mistake. Be careful. 
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Each time a teacher runs a consultation RC keeps a record of 
the skills the average male or female student has mastered on the 
data disk. When the original data disk is filled this record 
needs to be copied to a new data disk. To do this: 

1. Boot up the computer by placing the DOS disk in the A 
drive and turning it on. 

2. Remove the DOS disk. 

3. Place the original data disk in the A drive. 

4. Place the new data disk in the B drive (this is any 
blank formatted disk) . 

5. At the A prompt type in the following: 

A> copy A:*.??M B: (press return) 

then type in 
A> copy A:*.??F B: (press return) 

Use the new data disk to run additional RC consultations. 



Diskcopv 

Command 

A> diskeopy a:b: (press return) 



Function 

Copies the contents of the 
disk in drive A to the 
disk in drive B. 



Command Function 
A> 6ir>prn (press return) Sends directory to the 

printer rather than to 
the screen. 

A> type filename. log>pra (return) Sends the specific log 

file to the printer. 

A> type filename. rpt>pra (return) Sends the specific report 

file to the printer. 



Screen Displays 

Command Function 

A> type filename.log (press return) Ssnds the specific log 

file to the screen. 
A> type filename. rpt (press return) Sends the specific 

report file to the 

screen. 
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Appendix D 
Data Collection Instruments 
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Informed Consent 



Dear Teacher: 

During the next two years the Black Hills Special Education 
Cooperative and Granite School District will be cooperating with the 
Developmental Center for Handicapped Persons at Utah State 
University in a research project to investigate the special 
education referral and placement process. To conduct the research, 
it will be necessary to collect data on characteristics of students 
referred for assessment and the teachers. In addition, similar data 
will be collected on a sample of students not referred. During the 
first year, the data will be analyzed to determine what student and 
teacher characteristics are associated with referral. This analysis 
will provide valuable information about the referral process. In 
addition, a computer program will be developed to assist teachers in 
making referrals. During the second year the computer program will 
be used by teachers at selected schools. Comparisons will be made 
between schools that have the computer program to assist teachers 
and those that don't. The comparisons will allow for determining 
how helpful the program is to teachers and other school pei - onnel . 

It is anticipated that the results of the research will have 
the following benefits: 

1. Adding significantly to our knowledge about the special 
education referral process, 

2. Providing teachers and other school personnel with a 
validated computer program to assist in the referral process, 

3. Saving schools time and money by decreasing the number of 
students referenced for assessment. 

For this research to be successful it is important that as many 
teachers as possible participate. All that is required is that 
teachers fill out a brief information sheet (copy attached) . 
Identification numbers will be substituted for names of students and 
teachers to maintain confidentiality. In addition, all information 
will be held in strictest confidence by the research staff at Utah 
State University. 

If you are willing to participate please sign the lover portion 
of this form and return it in the enclosed postage paid envelope to 
the research project staff. 

Thank you. 



Signature Date 
school: 



Grade Taught: 

Return to Marilyn Likins at 2528 Beverly Street, SLC, UT 84106 
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Taachar Questionnaire 



Please respond to the questions below and return this 
questionnaire in the enclosed postage paid envelope to the 
research project staff. Responses will be Kept in strictest 
confidence. 

1. Name: 'Age: 

2 . School : 



3. Sex (circle one): Male Female 

4. Ethnic Background: 

White 

Black 

Hispanic 

Asian 

Other (please specify) : 



5. Marital Status (circle one) : Single Single-Parent 

Married Married-Parent 

6. Years of teaching experience: _ 

7. How many mainstreamed students with handicaps are in your 
class? 

8. How competent do you feel the professionals who receive 
referrals for special education assessment at your school are 
(circle one)? 

1 2 3 4 5 

Complete y Extremely 
Incompetent Competent 

9. How much encouragement to refer students for special 
education assessment have you been given by the professionals 
who receive referrals at your school? 

1 2 3 4 5 

No A Great Deal 

Encouragement of Encouragement 

10. The training you have received as a teacher enables you to 
cope adequately with low-IQ children in your class. 

1 2 3 4 5 

Disagree Agree 

11. The training you have received as a teacher enables you to 
cope adequately with behavior disordered children in your 
class. 

1 2 3 4 5 

Disagree Agree 
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12. 



A low-IQ child is less likely to be adequately prepared for 
life if ha or she is placed in a special class, rather than 
remaining in your class. 



1 2 3 4 5 

Disagree Agree 

13. A behavior disordered child is less likely to be adequately 
prepared for life if he or she is placed in a special class, 
rather than remaining in your class. 

1 2 3 4 5 

Disagree Agree 

14. The later a low-IQ child is removed from your class and 
placed in a special class, the greater will be the benefits 
to the child. 

1 2 3 4 5 

Disagree Agree 

15. The later a behavior disordered child is removed from your 
class and placed in a special class, the greater will be the 
benefits to the child. 

1 2 3 4 5 

Disagree Agree 

16. Low-IQ children who attend a special class will be better 
adjusted socially than children of equal intellectual 
ability who remain in your class. 

1 2 3 4 5 

Disagree Agree 

17. Behavior Disordered children who attend a special class will 
be better adjusted socially than children of equal 
intellectual ability who remain in your class. 

1 2 3 4 5 

Disagree Agree 

18. The presence of low-IQ children in your class does not make 
undue demands on your time. 

1 2 3 4 5 

Disagree Agree 
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19. The presence of behavior disordered children in your class 
does not make undue demands on your time. 



1 2 3 4 5 

Disagree Agree 

20. The presence of low-IQ children in your class impedes the 
progress of other children. 

1 2 3 4 5 

Disagree Agree 

21. The presence of behavior disordered children in your class 
impedes the progress of other children. 

1 2 3 4 5 

Disagree Agree 

22. The lack of equipment and materials available to you in your 
school does not enable you to cope effectively with low-IQ 
children in your class. 

1 2 3 4 5 

Disagree Agree 

23. The lack of equipment and materials available to you in your 
school does not enable you to cope effectively with behavior 
disordered children in your class. 

1 2 3 4 5 

Disagree Agree 

24. Low-IQ children who attend special classes will not attain 
as high a level of academic achievement as children of equal 
intellectual ability in your class. 

1 2 3 4 5 

Disagree Agree 

25. Behavior disordered children who attend special classes will 
not attain as high a level of academic achievement as 
children of equal intellectual ability in your class. 

1 2 3 4 5 

Disagree Agree 
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26. Mainstreaning is appropriate for students with mild and 
moderate handicaps. 



Disagree 



S 

Agree 



27. Mainstreaning is appropriate for students with severe 
handicaps. 



Disagree 



2 



5 

Agree 



28. For each student in your class please respond to the 
following questions. 



Studtnt 
Number 



Doe* this child usually 
need considerable teacher 
supervision during class 
activities? (circle one) 



Please estimate 

the child's 
academic potential 



How easy is thi 
chi Id to teach 





| 1 


2 


3 


*• 


5 
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Potential 


| to teach 
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Student 
Nuaeer 



Beat this child usually 
need considerable taaehar 
supervision during class 
ectivttlesT (circle one) 



Hesse sstiaste 

tha child'* 
ecadeaie potential 



Rom easy it thia 
child te teach? 



14 
15 
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1 2 5 4 5 6 
Msxiaua nfnlaua 
Suptrvision supervision 



1 2 5 4 5 6 
Lew High 
Potential Petentiei 



1 2 5 4 5 6 
Difficult Essy to 

to teach teach 
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1 2 5 4 5 6 
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1 2 5 4 5 6 
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1 2 5 4 5 6 



1 2 3 4 5 6 



1 2 5 4 5 6 
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1 2 5 4 5 6 



1 2 5 4 5 



1 2 5 4 5 6 
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1 2 5 4 5 6 



1 2 5 4 5 6 



1 2 5 4 5 6 
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1 2 3 4 5 6 



1 2 5 4 5 6 



1 2 3 4 5 6 



1 2 3 4 5 6 
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1 2 3 4 5 6 



1 2 3 4 5 6 



1 2 3 4 5 6 



1 2 3 4 3 6 



1 2 5 4 5 6 



1 2 3 4 5 6 



1 2 5 4 5 6 



1 2 5 4 5 6 



1 2 5 4 5 6 



1 2 5 4 5 6 



1 2 3 4 5 6 



1 2 5 4 5 6 
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Referred student Data 



ISAShSX 
1« Name: 



2. School: 



3. Sax: Male Female (circle one) 

4. Ethnic background: 

White: 

Black: 

Hispanic: 

Asian: 

Other, please 

specif y:, 



5. Marital Status: Single Married (circle one) 
i^SS^SitS^ in the Class th * ^udent is 

J;f n L«! scale below please indicate how compe-.enr you feel the 
the ^ assessment team the student is being referred to is 7 circle 

1 2 3 4 5 * 

Completely * , 

1 2 3 4 5 « 

Hot in favor „„, ? „ 

M.i».tr.„i„ g a&as 

1 2 3 4 5 6 

Very Low „ „. . 



» 81 



10. On the seals below please indicate how variable the ability 
level of the class the student being referred from is. circle 
ones 

1 2 3 4 5 6 

I*ow High 
variability Variability 



Student Bftjfca 

11. Identification Number: 



12. Sex: Male Female (Circle one) 

13. Ethnic Background 

White 

Black 

Hispanic 

Asian 

Other, please 

specify 

14. Was this student placed in special education? Yes No 

(Circle one) 



75 82 



Control Student Data 



Taacher £a£&L 
X. Nana? 



2. School: 



3. Sex: Male Female (circle one) 

4. Ethnic background: 

White: 

Black: 

Hispanic: 

Asian: 

Other, please 

specify : 



5. Marital status: single Married (circle one) 

6. How many ma Instretmed rtudentj are in the class the student is 
being referred from? 

7. one the scale below please indicate now competent you feel the 
the assessment team the student is being referred to is. Circle 
one: 

1 2 3 4 5 6 

completely Extremely 
Incompetent Competent 

8. One the scale below please indicate how you feel about 
mainstreaming students with handicaps, circle one: 

1 2 3 4 5 6 

Not In favor Very In Favor 

Mainstreaming Mainstreaming 

9. On the scale below please indicate the ability level of the 
class the student is being referred from. Circle one: 

1 2 3 4 3 6 

Very Low Very High 

Ability Ability 
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10. On the scale below please indicate how variable the ability 
level of the class the student being referred from is. circle 
ones 

rL 2 3 4 5 6 

Low H4 . 

variability Variability 

Student: Data 

11. Identification Number: 



12. Sex: Male Female (Circle one) 

13. Ethnic Background 

White 

Black 

Hispanic 

Asian 

Other, please 
specify 



77 84 



